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Summary of research 

The goal of this project was to measure how neurons in the superior colliculus (SC) change 
their activity during a visual search task. Specifically, we proposed to measure how the 
activity of these neurons was altered by the discriminability of visual targets and to test 
how these changes might predict the changes in the subjects’ performance. The primary 
rationale for this study was that understanding how the information encoded by these 
neurons constrains overall search performance would foster the development of better 
models of human performance. 

Work performed during the period supported by this grant has achieved these aims. 

First, we have recorded from neurons in the superior colliculus (SC) during a visual search 
task in which the difficulty of the task and the performance of the subject was 
systematically varied. The results from these single-neuron physiology experiments shows 
that prior to eye movement onset, the difference in activity across the ensemble of neurons 
reaches a fixed “threshold” value, reflecting the operation of a “winner-take-all” 
mechanism. 

Second, we have developed a model of eye movement decisions based on the principle of 
“winner-take-all”. The model incorporates the idea that the overt saccade choice reflects 
only one of the multiple saccades prepared during visual discrimination, consistent with 
our physiological data. The value of the model is that, unlike previous models, it is able to 
account for both the latency and the percent correct of saccade choices. 


Background 

Human performance is easily impaired in a variety of aerospace tasks - such as remotely 
docking a spacecraft, or flying an aircraft in poor visibility, or finding and tracking a particular 
target airplane on a crowded air-traffic control display. These impairments occur because 
aerospace environments can degrade or remove the sensory cues used to drive and coordinate 
voluntary actions, and display systems may not be optimally matched to the human user. 
Fortunately, much progress has been made recently in the field of neuroscience toward 
understanding the physiological mechanisms responsible for voluntary motor behaviors. In 
particular, we are beginning to understand how the brain accomplishes visual search - the 
process that allows us to quickly respond to important items in our surroundings, even in the 
presence of other irrelevant stimuli. Understanding how individual neurons participate in this 
process is critical for identifying the factors that limit the speed and accuracy of visual search. 
Knowledge of these neural factors could be used to design visual displays that conform to the 
temporal and spatial constraints imposed by human neurobiology, thereby directly improving 
human performance in man-machine interfaces and virtual environments. 

Arguably the most important class of movements made by primates during visual search 
is eye movements, because control of the position of the eyes determines which visual 
information is sampled and for how long. Decades of anatomical and physiological research 
have provided a detailed outline of the brain pathways that are responsible for the control of eye 
movements in primates. These pathways involve a network of areas including visual areas of the 
cerebral cortex (which provide information about the content of the visual scene), motor regions 
of the cerebellum and brainstem (which form the final motor commands), and several regions 


that lie in-between (and which possess a mixture of sensory and motor properties). It is these 
latter, intermediate, brain regions that are most likely responsible for forming the decision about 
where the eyes should move next during visual search, because they lie at the interface between 
the visual areas that provide evidence about the location of stimuli and the motor regions that 
execute the movements. One of these structures, the superior colliculus (SC), has long been 
known to play a pivotal role in the control of saccadic eye movements. However, most studies of 
the SC have focused on its contribution to the motor execution of saccades. Because of the 
importance of the SC in the decision processes that precede saccade generation, examination of 
this structure during visual search is likely to reveal fundamental aspects of the visual neural 
computations involved in identifying, localizing, and selecting visual targets. 

Progress report 

1. Determining whether and how changes in SC neuron activity during a visual 
discrimination task is related to changes in subject performance 

The monkey performed a discrimination task while we recorded the activity of single 
neurons in the SC. To adjust the difficulty of the task, we used stimuli that were embedded in a 
background of visual noise (as shown in figure). By varying the luminance distribution of the 
background noise, and the luminance increments (pedestals) associated with the distractor and 
target stimuli, we presented stimuli of known and adjustable discriminability (d’). 

During the task, the monkey initially 
fixated a small target (fixation cross) located 
at the center of a background of visual noise. 

After a random interval (500-2000 ms), the 
target and distractor stimuli appeared. The 
stimuli consist of small regions in which 
luminance steps have been added to the 
background noise. The region with the 
largest step (in this case, the upper right 
region) was defined as the target. The spatial 
locations of the stimuli were arranged so that 
either the target or one of the distractor 
stimuli lay within the response field of the 
neuron. The background noise during the 
fixation and stimulus intervals was identical 
so that the appearance of the stimuli was not 
masked by changes elsewhere in the display. 

The monkey was rewarded for making a 
saccade to the target region. On interleaved trials, the difficulty of the task was varied by 
changing the discriminability associated with the target stimulus. Based on data obtained during 
preliminary behavioral training prior to single-neuron recording, we adjusted the stimuli to elicit 
performance ranging from chance to near perfect. On some trials, we provided a cue that 
indicated the likely location of the target (75% valid). 

We determined how the activity of SC neurons changed over time when the stimulus in 
its response field was a target or a distractor. We measured the activity of SC neurons before and 
after target and distractor onset, as well as during the interval corresponding to the saccade eye 




movement itself. We quantified the activity in these intervals by measuring average firing rate, 
and also by using an analysis based on signal detection theory, which involves the construction 
of “Receive Operating Characteristic” curves. This ROC analysis provides a measure of the 
relative firing rate, rather than the absolute firing rate. 

We found that in an interval prior to saccade initiation, the activity of single neurons 
reached a fixed ROC area across our signal strengths whereas the average firing rate over the 
same interval increased as a function of task difficulty. This finding argues that the criterion to 
initiate saccades is a firing rate difference. The presence of the cue in the response field of a 
neuron increased the target probability ROC area by a fixed amount. Valid cues both decreased 
the latency and increased the proportion of correct responses. These findings suggest that the 
superior colliculus might encode a decision signal as a difference in firing rate across locations, 
triggering a saccade when a criterion level of difference is reached. 


2. Developing a model of eye movement decisions based on “winner-take-all” principle 

Using the same task described above, we collected a large set of behavioral data, 
consisting of approximately 50,0000 saccades. This data set allowed us to thoroughly document 
two key aspects of performance in the task: 1) how the percentage of correct answers changes as 
a function of task difficulty (i.e., stimulus discriminability), and 2) how the distribution of 
reaction times varies as a function of task difficulty. 

Previous work on modeling the decision process for saccades have emphasized the 
distribution of reaction times, and have not attempted to account for the percentage of correct 
responses. For example, one of the best-known class of models, the linear rise-to-threshold 
models, is very successful at account for latency distributions, but does not account for the 
accuracy (% correct) of saccade choices. Based on our physiological data, we extended this 
model by incorporating the principle of “winner-take-all”. 

We modeled the decision process by applying a “max rule” to the rates drawn from two 
distributions, each representing one saccade goal. Applying this rule to our data involved 
incorporating predictions derived from probability theory to describe the distributions of saccade 
reaction times and the percentage of correct and incorrect responses. We found that the shapes of 
the correct and incorrect latency distributions as well as their relative proportions were well fit by 
applying this rule. As signal strength increased, the monkey’s proportion correct increased and 
saccadic latency decreased; rates associated with correct responses increased with signal strength 
and rates for incorrect responses remained constant. 

These results show that a model of eye movement decisions including a max rule can 
account for both the latency and percent correct of saccadic choices. The max rule incorporates 
the idea that the overt saccadic choice reflects only one of the multiple saccades prepared during 
visual discrimination. This type of model, based in large part on physiological data, may be 
useful in determining how saccadic decisions are affected by task parameters such as signal 
strength and prior probability. 



Significance: 


The results from these experiments provide new insights into the biological mechanisms 
underlying visual search behavior in humans and other primates. Although the immediate goals 
of the project were to expand our basic biological knowledge, the results are also be applicable to 
the stated goals of NASA’s Space Human Factors, Intelligent Systems, and Aerospace Operation 
Systems Programs. Biological systems are capable of accomplishing tasks that have proved 
extremely difficult to duplicate using artificial intelligence and robotic systems - especially the 
job of detecting, discriminating and responding to stimuli quickly and accurately, even in the 
presence of background noise. Our results have provided new data about the algorithms used by 
biological systems to accomplish such tasks, and may therefore aid in the development of robust 
computational models of human visuomotor performance and in the design of the autonomous 
systems for future aerospace applications (e.g. rovers for the planeterary exploration, robotics, 
and other biomimetic intelligent systems). Our development of a new model of eye movement 
decisions is a first step in that direction. 
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